Papers by Prudhvi Raj Dachapally
Query-Key Normalization for Transformers (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Low-resource language translation is a challenging but socially valuable NLP task. |
| Approach: | They propose a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. |
| Outcome: | The proposed technique improves 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT’15. |